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ABSTRACT 

Blended learning is already a strongly established way of teaching in higher education. In support of face-to-face 
teaching, e-courses may vary in structure, assignments, prompt examinations, interaction between students and teachers 
etc. In the paper, we present an analysis of factors that influence the usefulness of e-courses as perceived by students. A 
survey was conducted among students of the public administration programme at the Faculty of Public Administration, 
University of Ljubljana. Students evaluated 13 different aspects of e-courses in which they were enrolled. These aspects 
were assessed on a 7-point opinion scale. Results obtained in the survey were connected with certain demographic data, 
such as gender, high school final grade, year of study, study programme etc. A multiple linear regression was used with 
perceived usefulness as the response (dependent) variable and the 12 other e-course aspects as predictors (independent) 
variables. Further, the same regression analysis was performed on different subgroups of students based on 
demographical data and analysed where the impacts differ from the results for the whole sample. The empirical results 
showed that the general impression regarding the e-courses, their consistency with the face-to-face teaching and the 
teacher’s responsiveness had a significant influence on the students’ perception of the usefulness. On the other hand, the 
structure of an e-course and the variety of different assignments did not have a significant impact on perceived 
usefulness. Further analysis based on demographic data revealed several interesting subgroups of students where the 
perception of usefulness was influenced by completely different aspects compared to the results for the whole sample. 
The results may help teachers with managing the content and structure of an e-course and, as assumed, improve the 
perceived usefulness of blended learning for different subgroups of students. 
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1. INTRODUCTION 

In the last two decades, the use of information and communications technology (ICT) in the education 
process has triggered many changes in teaching approaches and techniques. Teachers therefore have new 
possibilities to make the pedagogical process more interesting and interactive. They can offer different 
assignments and ways of communicating without limits (regarding the place and time of teaching and 
learning). In this way, they go along with the new generations of students that were ‘born with smartphones’ 
and are very familiar with the latest technologies. According to Jones et al. (2010) and Kubiatko (2013), 
today's young generation has a different way of thinking and functioning to previous generations. 

E-learning involves the use of ICT to deliver teaching and learning and is becoming increasingly 
important in higher education (Penny, 2011). When talking about changes in higher education, the 
introduction of new ICT tools into the teaching and learning processes has to be successful in the eyes of all 
stakeholders - students, teachers and management of the higher education institution. Based on adequate 
measurements, statistical analyses of the consequences of implementing a learning management system 
(LMS) have to be performed regularly and improvements made in order to find out whether the results are 
acceptable or if any improvements are needed. When deciding on the introduction of an LMS and when 
measuring its effectiveness and usefulness, many factors have to be taken into account. Especially on the 
teachers’ side, many doubts emerge when a new ICT is in question. Many aspects have to be considered, 
especially when deciding whether to completely replace face-to-face learning with e-learning in a selected 
LMS or to choose blended learning where the virtual classroom supplements lectures in traditional 
classrooms. The students’ point of view is also important since they are the main users of the offered 
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teaching technique and can cooperate in implementing and improving an e-course as a very important 
stakeholder in the e-learning process. Therefore, both teachers and students should be regularly asked to give 
their opinion on the work and their feelings in the virtual classroom. 

In our study, we analysed only the students’ point of view on the courses in which they were enrolled. We 
examined the results received from students of the two undergraduate programmes at the Faculty of Public 
Administration, University of Ljubljana where LMS Moodle is used for e-learning. The purpose of the paper 
is thus to present an analysis of the factors that influence the usefulness of an e-course as perceived by the 
students. The paper explains how the students evaluate the usefulness of e-courses at the faculty level and 
discusses the variances among the different subgroups of students. 

The paper is structured as follows: after the introduction, a brief literature review on various aspects of 
blended learning and its usefulness through a presentation of previous work in the field is given. The third 
part includes a presentation of the empirical research. At the end, based on the examined data, interpretations, 
conclusions and suggestions are offered. They are accompanied by the paper’s limitations and avenues for 
further research. 


2. LITERATURE REVIEW 

The impact of the Internet on education at all levels has captured both teachers’ and students’ attention in 
recent years (Elkaseh et al., 2016). In some countries, the use of ICT in schools at all levels of education has 
been strongly supported by the government through initiatives for primary schools through to higher 
education. They have encouraged the acquisition of laptop computers for students and teachers with 
favourable conditions and secured broadband connections in all public establishments (Carvalho et al., 2011). 

With the ever present need to demonstrate value for money and maximise efficiency and effectiveness 
from training and development within an often restricted time and expenditure framework, measuring the 
impact of enhancing knowledge management by using technology is of constant interest and importance 
(Smedley, 2010). The assessment of whether the implementation of an e-course is successful or not should be 
given by both groups of users, teachers and students. Of course, the management of the educational 
institution also has to discover if the introduction of blended learning leads to better student results (higher 
grades, fewer admissions to exams), lower costs and the satisfaction of all stakeholders (Hall, 2006; Kohang 
and Durante, 2003; Upadhyaya and Mallik, 2013, Yukselturk & Bulut, 2007). 

Although many universities across the world have incorporated Internet-based learning systems, the 
success of their implementation requires an extensive understanding of the end-user acceptance process 
(Al-Adwan et al., 2013). Saade et al. (2007) point out that “in general, like any information system, user 
acceptance and usage are important primary measures of system success”. The user perspective is therefore 
crucial to examine in the implementation of an LMS and to evaluate its success (Hall. 2006). Zivic-Butorac 
et al. (2001) claim that students’ perception of e-learning is one of the most important steps in developing 
and implementing a successful e-learning environment. Hrastinski (2009) provides a review of the literature 
in the area of online learner participation, and claims that participation and learning are intricately 
interrelated and that, in order for learners to take full advantage, the participation experience needs to be 
satisfactory. 

To understand the importance of the students’ point of view, in our study we focused only on them. 
According to Francis and Raftery (2005), we examined the usefulness of blended learning, whereas there are 
two other modes of e-learning engagement (1. baseline course administration and learner support, and 2. a 
fully-fledged online course). Blended learning is a combination of traditional face-to-face teaching and an 
online course. This mixes the features of virtual and real environments to provide for the holistic production 
of information and enhance students’ learning experience (Al-Adwan et al., 2013). 

When talking about blended learning, the successful implementation of e-course, in which students and 
teachers cooperate in both a virtual and classical classroom, depends on many factors. Vonderwell and 
Zachariah (2005) claim that online learner participation is influenced by technology and interface 
characteristics, content area experience, student roles and instructional tasks, and information overload. 

The open-source Moodle LMS can be found in many segments of education and higher education is no 
exception. Its popularity, except for the fact it is free, is mainly based on its flexibility, adaptability and the 
possibility of personalisation while, on the other side, the system contains many standard features which 
make the learning process easy to implement. As Liao et al. (2011) stated: “Moodle e-learning platform is 
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easy to use and provides a good communication tool, discussion area, group space, workspace, and makes 
learning more interesting”. When comparing some open-source LMSs, Kareal and Klema (2006) concluded 
that Moodle is one of the most adaptable systems and highlighted this feature as a significant element of an 
effective e-learning system. They also stressed that Moodle is the most user-friendly e-learning system 
among the systems under comparison. Carvalho et al. (2011) studied students’ perceptions of Blackboard and 
Moodle at Portuguese universities and ascertained that all their findings “reveal a student preference towards 
Moodle”. In their research, they also found that e-learning materials substituted the traditional courses. 

The Faculty of Public Administration implemented blended learning via the Moodle LMS platform in the 
2010/11 academic year. After three years of gradually introducing them, e-courses became mandatory for all 
undergraduate study courses (for more, see Umek et al., 2015). Until now, quite a few changes have been 
made and some additions introduced in virtual classrooms. Currently, each lecture is supported by e-content 
followed by a quiz to check understanding of the prepared content in the e-course. For the tutorial, two 
extensive classroom assignments are prepared during the semester and the teacher gives feedback on the 
correctness of the solutions. 

In our study, we determined 12 factors which we assumed influence students’ perceptions of the 
usefulness of the Moodle LMS. They are related to organisation of the e-course (goals, materials and 
assignments), the lecturers’ activities (assessments, responses), the students’ preferences regarding learning 
online or in the classroom, the general impression of the e-course and the degree of consistency with the 
lessons in the classroom. We believe that the study presented below contributes some important findings to 
both theory and practice in the field of blended learning. 


3. EMPIRICAL RESEARCH 
3.1 Methodology and Data 

In our study, we analysed how different aspects of an e-course influence students’ perception of the 
usefulness of blended learning. For this reason, we developed a questionnaire and asked students from the 
Faculty of Public Administration, University of Ljubljana to participate in the survey. We limited our survey 
to courses that were obligatory for undergraduate students. Blended learning is mandatory for these courses. 

The questionnaire consisted of 13 statements (Table 1) describing the e-course and some other questions 
about the students (high school final grade, occupation with other activities outside of study etc.). The 
students expressed their opinion regarding the statements in Table 1 on a 7-point Likert scale from “totally 
disagree” (value 1) to “totally agree” (value 7). The statements were based on empirical findings from recent 
literature (Upadhyaya and Mallik, 2013, Zivic-Butorac et al. (2001)). Since the primary focus of our study 
was to analyse factors which influence students’ perceived usefulness, statement Q13 was chosen as the 
dependent variable. 

Table 1. Statements from our survey with abbreviations Q1-Q13 

Abb. Statement about e-course 

Q1 The virtual classroom of the course is organised transparently. 

Q2 The goals (workload demands, grading) of this e-course were clearly stated at the start of the semester. 

Q3 This e-course offers a variety of ways to assess my learning (quizzes, written work, forums, files, . . .) 

Q4 I receive the teacher's comment/feedback on an assignment in less than 7 days. 

Q5 I prefer fewer lectures in the traditional way (face-to-face) and more learning material processed in the e- 

course. 

Q6 More tutorials in the course could be carried out in the e-course instead of in the classroom. 

Q7 My general impression of the e-course is good. 

Q8 The study material and tasks of the e-course are presented in a clear and understandable way. 

Q9 Finding certain activities in the e-course is simple. 

Q10 The prepared learning material and tasks are consistent with the lectures in the classroom and supplement 
them. 

Q1 1 The prepared material and assignments supplement the tutorial in the classroom. 

Q12 The teacher gives me feedback/a response on my submissions (assignment, forum posts). 

Q13 Learning materials and activities in the e-course helped me to effectively study this subject matter. 

Source: Survey, 2015 
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The questionnaire was completed by 315 students, with each student evaluating 3 to 5 different e-courses. 
We received a total of 1,456 e-course evaluations. Due to missing values, we removed some records from our 
initial data set. The final sample for analysis is contained in 1,083 records, each representing a student 
evaluating one e-course. We then added certain demographical data (gender, year of study, study programme 
etc.) to our final data set. Our goal was to relate the students’ perceived usefulness of blended learning 
(statement Q13) with the other variables from Table 1. We built a multiple linear regression model with 
variable Q13 as the response variable and the others (Q1-Q12) as predictors. 

In the empirical results part which follows, we report the coefficients, unstandardized and standardized, 
and the corresponding p-values. We highlight and discuss variables showing a significant impact on the mean 
perceived usefulness of blended learning. For the level of significance, we chose a = 0.05. 

Our recent studies (Umek et al., 2015) have already shown the heterogeneity of students at the Faculty of 
Public Administration. Regression analysis on the whole sample could therefore hide interesting patterns in 
our data set. For this reason, we repeated the analysis on different subgroups of students based on their 
demographic characteristics. For convenience, in the subgroup analysis we report just the predictor variables 
with a significant impact and do not report the p-values. Due to the high number of tested hypotheses, we 
used a Bonferroni correction to control the familywise error rate. 


3.2 Empirical Results 


The multiple linear regression showed that variables Q1-Q12 have a significant impact on students’ mean 
perceived usefulness (Fi 2i io 7 o= 106.065, p < 0.001). The model explained 54.3% of the variance in students’ 
perceived usefulness, while the standard error of the estimate was 1.039. Table 2 shows coefficients 
(unstandardized and standardized), t-statistics and the corresponding p-values for all predictor variables 
Q1-Q12. Predictors with a significant impact on the mean value of Q13 (usefulness) are shown in bold. 

Table 2. Parameters of the linear regression model with the perceived usefulness of blended learning as the response 

variable 



Unstandardized 

Coefficients 

Standardized 

Coefficient 



Variable 

B 

Std. Error 

Beta 

t 

Sig. 

intercept 

-0.541 

0.185 


-20.922 

0.004 

Qi 

-0.002 

0.043 

-0.002 

-0.047 

0.963 

Q2 

-0.024 

0.039 

-0.021 

-0.605 

0.545 

Q3 

0.037 

0.035 

0.032 

10.062 

0.288 

Q4 

-0.093 

0.026 

-0.095 

-30.554 

0.000 

Q5 

0.123 

0.024 

0.168 

50.096 

0.000 

Q6 

-0.015 

0.024 

-0.021 

-0.623 

0.534 

Q7 

0.252 

0.042 

0.222 

60.033 

0.000 

Q8 

0.151 

0.040 

0.131 

30.814 

0.000 

Q9 

0.010 

0.018 

0.012 

0.548 

0.584 

Q10 

0.252 

0.042 

0.215 

60.018 

0.000 

Qll 

0.210 

0.041 

0.181 

50.155 

0.000 

Q12 

0.180 

0.030 

0.164 

60.035 

0.000 


The strongest impact on perceived usefulness was found for variable Q7 (good general impression of the 
e-course) which has the largest standardized regression coefficient of 0.222. This means that an increase in 
the general impression of an e-course by 1 standard deviation on average increases the perceived usefulness 
by 0.222 of a standard deviation. 

The other predictors with a very similar (yet somewhat weaker) impact on response Q13 are Q10 and Q 1 1 
(e-course supplements face-to-face learning well), Q7 (general impression of the e-course), Q5 (preference 
for e-learning over face-to-face learning), Q12 (teacher’s response), Q8 (understandability of the study 
material) and Q4 (teacher’s prompt feedback). The impact of all mentioned predictors (with the exception of 
Q4) is positive, i.e. an increase in them on average increases students’ perceived usefulness. 
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On the other hand, the impacts of the other predictors are not significant. These predictors are Q1 
(transparent structures of the e-course), Q2 (workload demands clear in advance), Q3 (a variety of activities), 
Q6 (e-courses instead of a face-to-face tutorial) and Q9 (simplicity of finding relevant activities). 

The empirical study of the whole sample showed that a good general impression of the e-course and 
supplementing ttaditional face-to-face learning with e-learning increase students’ perceived usefulness of 
e-course. It means that an e-course which offers additional topics to face-to-face learning and makes a good 
general impression on the students is more likely to be perceived as useful by students. 

As already mentioned, our previous study (Umek et al., 2015) revealed that the student population at the 
Faculty of Public Administration is very heterogeneous. We showed that the introduction of the Moodle 
LMS in recent years has had different impacts on various subgroups of students. Therefore, we expected that 
the perceived usefulness of blended learning could also be influenced by different factors among various 
subgroups of students. We applied the same linear regression model (response variable Q13, predictors 
Q1-Q12) to 15 subgroups of students based on their demographic characteristics. These subgroups were 
established based on their gender, high school final grade (four groups: from sufficient (final grade 2 out of 
5) to excellent), year of study (three groups, one for each year), study programme (two groups: university, 
professional), region (the nine regions of Slovenia from where students come) and their occupation with 
other regular activities besides study per week (six groups based on the amount of student work, sports 
training etc.). Our preliminary study suggested analysing just two ‘regions’ (students from the Ljubljana 
region where the faculty is located, and all other regions). Due to transparency, we reported just two 
subgroups of students based on their other activities (no other activities, more than 6 hours of extracurricular 
activities per day). 

Table 3 shows the results of the linear regression on the 15 analysed subgroups, where each row presents 
the results for a particular subgroup. The first two values are the size of the subgroup (n) and its coefficient of 
determination (R 2 ) of the related linear model, followed by the impacts of predictors Q1-Q12, indicating 
factors with a significant impact on the mean perceived usefulness. Instead of reporting the p-values, we 
divided the impacts into four categories, from non-significant with an empty cell to highly significant with 
***). The table allows us to see how the analysed subgroups differ (which factors are important for 
usefulness in which subgroup). 

Table 3. Linear models on different subgroups of students. We report their sizes (n), coefficients of determination (R 2 ) 
and indicate predictors Ql- Q12 with a significant impact on the mean of Q13 



Subgroup 

n 

R 2 

Ql Q2 Q3 Q4 

Q5 

Q6 

Q7 

Q8 

Q9 Q10 

Qll 

Q12 


whole sample 

1083 

0.543 

* 

*** 


*** 

* 

*** 

*** 

*** 

Gender 

male 

284 

0.514 




* 




* 


female 

799 

0.560 

* 

** 


** 

* 

*** 

** 

*** 

High school 

sufficient 

442 

0.530 

* 

*** 


* 

* 

* 


** 

final grade 

good 

318 

0.568 






*** 


* 


very good 

205 

0.570 










excellent 

113 

0.724 




* 



*** 


Year of 

1st 

698 

0.527 


* 


*** 

* 

* 

*** 

** 

study 

2nd 

161 

0.556 




* 






3rd 

224 

0.598 






** 


*** 

Programme 

university 

436 

0.564 

* 



* 


* 


*** 


professional 

647 

0.549 


** 


*** 

* 

*** 

** 


Region 

outside Ljubljana 

473 

0.600 






** 

* 



Ljubljana 

610 

0.525 


** 


*** 

* 

** 

** 

*** 

Other 

no activities 

71 

0.522 


** 

** 






activities 

more than 6 hours per day 

289 

0.601 


* 


** 




* 


Legend: 


empty cell Bonferroni adjusted a > 0.05 
* 0.01 < a <=0.05 

** 0.001 < a <=0.01 

*** a <=0.001 
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Table 3 shows that the interpretation of the results is the same for the whole sample and for female 
students. The predictors with a significant impact on perceived usefulness are the same. On the contrary, only 
two aspects are important for the male university students: the general impression of the e-course and the 
teacher's responsiveness. The analysis showed differences among students based on their high school grades; 
students with the best high school grades find the usefulness of blended learning in supplementing the 
tutorials. In contrast, students with lower high school grades see the usefulness in the replacement of 
face-to-face learning. 

The factors influencing perceived usefulness also change when comparing the year of study. In higher 
years of study, the teacher’s feedback and supplementing the tutorials have an influence on perceived 
usefulness. On the other side, the general impression of the e-course does not play a significant role any 
more. Table 3 also shows that in the first year of study many aspects are significant whereas later only a few 
of them remain important. 

Table 3 shows differences between the two study programmes, the amount of other activities outside of 
study and the region of Slovenia where students come from. Students who live away from the university 
campus (region: outside Ljubljana) find the usefulness of blended learning in supplementing traditional 
learning, both tutorials and lectures. For this subgroup, only two aspects have a significant impact. 

Although our sample is very mixed, some characteristics of all analysed subgroups are the same. We 
found no significant impact on perceived usefulness for predictors Q1 (transparent structures of the e-course), 
Q2 (workload demands clear in advance), Q3 (a variety of activities) and Q9 (simplicity of finding relevant 
activities). 

To clarify our study, we presented the results from Table 3 in the form of a network (Figure 1). The 
network consists of two sets of nodes which represent the analysed subgroups (black dots) and statements 
from the questionnaire (independent variables, grey dots). Two nodes are connected if a variable has a 
significant impact on perceived usefulness of the analysed subgroup. Note that some of the analysed 
subgroups and statements are not shown in the network since they do not reflect any significant findings. Our 
network was plotted using the Pajek software (Batagelj & Mrvar) with a Fruchterman-Reingold (Fructerman 
& Reingold, 1991) drawing algorithm. 



Figure 1. Network of variables and subgroups. A subgroup (black dot) and a variable (grey dot) are connected if we 
found a significant impact of the variable on the perceived usefulness of blended learning in a subgroup of students 

We can clearly see in the network in Figure 1 which vertices have many connections. If a grey node is 
connected to several subgroups it means that the variable has a significant influence on perceived usefulness 
for different subgroups of students. We can deduce such statements from Figure 1: general impression, 
teacher’s feedback, clarity of study materials etc. On the contrary, the nodes with less connections are 
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specific to fewer subgroups. Some statements from the questionnaire are missing, which means we did not 
find any subgroup where they play an important role regarding perceived usefulness. 

Similar conclusions can be drawn for the black dots. If a subgroup is connected with several variables it 
means there are several factors which influence the students’ perceived usefulness. Examples of such 
subgroups are female students, students from the Ljubljana region, students with a sufficient high school 
grade etc. In contrast, for students in the second year of study just the general impression plays a significant 
role. There are some other subgroups with only two influencing factors, such as male students, students with 
excellent high school grades etc. Since various subgroups are connected to different variables, Figure 1 
clearly shows the heterogeneity of the population we analysed. 


4. CONCLUSION 

The main goal of the paper was to present an analysis of the factors that influence the usefulness of an 
e-course as perceived by students. We examined results received from students of two undergraduate 
programmes at the Faculty of Public Administration, University of Ljubljana where LMS Moodle is used for 
e-learning. In this respect, the paper’s key contribution is explaining how the students evaluate the usefulness 
of e-courses at the faculty level and the discussion of the variances among the different subgroups of 
students. 

In order to confirm theoretical expectations, we empirically analysed the influence of 12 aspects of an 
e-course. Four of them (structures of e-course, workload demands being clear in advance, a variety of 
activities and simplicity of finding relevant activities) had no significant impact on perceived usefulness - on 
either the whole sample or on each subgroup analysed. Our empirical results also showed that the general 
impression regarding the e-course, their consistency with face-to-face learning and the responsiveness of 
teachers had a significant positive influence on the students’ perceived usefulness. More detailed analysis 
revealed interesting subgroups of students where the relationship between perceived usefulness and the other 
analysed aspects differs. We discovered that the general impression plays an important role in determining a 
higher level of perceived usefulness in the majority of subgroups we examined. For male students, besides 
the teacher’s responsiveness, the general impression was the only significant aspect in relation to perceived 
usefulness. However, we discovered that the general impression loses its impact in higher years of study 
when the other aspects become important (teacher’s feedback, supplement to face-to-face learning). 

Interestingly, the factor “transparent organization of the virtual course” was non-significant in all 
subgroups we analysed. On the contrary, the intuitively similar “general impression” factor played an 
important role in almost all subgroups. The results therefore reflect the fact that “transparent organization” 
and “general impression” measure two different aspects - the e-classroom can be transparent but its general 
impression may still be poor. Such courses probably only provide basic information in basic colours and 
fonts without any additional pictures, links to multimedia sources etc. In contrast, some students may prefer a 
less organized structure with lots of colours in the title, funny pictures etc. Accordingly, these findings may 
provide useful guidelines for the structure and design development of e-courses in the future e-learning 
system at the Faculty of Public Administration. 

However, as we focussed on the perceived usefulness of blended learning from the students’ perspective, 
our study obviously neglects the teachers’ perspective. To overcome this limitation, we plan to develop a new 
survey. We will ask teachers about their views on blended learning (amount of work needed for an e-course, 
communication with students, preferences (e-courses vs. face-to-face courses) etc.). In the next step, we will 
aggregate the current results to the level of an e-course and link them with the results emerging form the new 
questionnaire. Moreover, our recent study showed a significant increase in students’ performance in the 
period after the Moodle LMS was introduced. Therefore, in our future research we plan to use data on 
students’ performance and link the results with our current study. This extended study would indicate which 
aspects of e-courses are linked to better student performances. 
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